Incrementally Identifying Objects from Referring Expressions using Spatial Object Models

An important problem in human-robot interaction is that of referring expressions: phrases used to identify a particular object among others. Existing parsing models all operate on entire sentences. Incrementally parsing referring expressions is important for human-robot interaction and conversational feedback. We present a model for parsing real-word referring expressions, trained and tested on human-provided data. In our test corpus, when presented with the entire sentence, our model ranks the correct object as the most likely 60.3% of the time, and ranks the correct object in the top three 79.0% of the time. Given the entire sentence humans identify the correct object 79.0% of the time. With 50%, 80% and 90% of the sentence, the model ranks the correct object as the most likely 17.4%, 27.8%, and 36.2% of the time. Our parser is capable of keeping up with human speech, with 80% of all words processed on commodity hardware within 10ms, and 95% of all words in about 300ms.